Ad Hoc Software Testing

An ad hoc test can be described as an exploratory case that you expect to run only once, unless it happens to uncover a defect. As such, ad hoc testing is sometimes viewed as a wasteful approach to software testing. On the other hand, many skilled software testers find the exploratory approach to be one of the best techniques for uncovering certain types of defects. In this paper we’ll put ad hoc tests into perspective with other forms of exploratory testing. Along the way, we’ll also develop a comparison between jazz improvisation in the musical world, and improvisational software testing. Introduction You might wonder why we’re placing such emphasis on the idea that ad hoc tests are fundamentally one-off cases. To begin with, the one-off characteristic distinguishes ad hoc tests cleanly from the formal testing paradigms that place the emphasis on re-executing tests, such as acceptance testing and regression testing. The assertion that one-off tests are valuable flies in the face of conventional wisdom, which places such a premium on repeatability and test automation. Much of the software industry overemphasizes the design and rerunning of regression tests, at the risk of failing to run enough new tests. It is sobering to ask why we are spending so much time rerunning tests that have already passed muster, when we could be running one of the many worthwhile tests that have never been tried even once? In fairness, we understand that these issues are highly contextual. For example, if you’re working with life-critical software, there are important ethical and legal reasons for placing such a heavy emphasis on regression testing. In general, though, we believe the software industry can benefit by striking a better balance between one-off tests, and those that are designed to be rerun. We’ll return to this theme in the next section (Ad hoc testing versus regression testing), but first, let’s take a look at what has already been written on this topic. For various reasons, many people still associate ad hoc testing with an aimless black box approach. In practice, skilled testers use methods that are neither aimless, nor restricted to black box methods. The old notion of ad hoc testing is currently enjoying a metamorphosis into the more highly evolved approach called Exploratory Testing. We first came upon the term Exploratory Testing in the original edition of Testing Computer Software (Kaner, 1988). Here are a few excerpts: “At some... point, you’ll stop formally planning and documenting new tests until the next test cycle. You can keep testing. Run new tests as you think of them, without spending much time preparing or explaining the tests. Trust your instincts... In this example, you quickly reached the switch point from formal to informal testing because the program crashed so soon. Something may be fundamentally wrong. If so, the program will be redesigned. Creating new test series now is risky. They may become obsolete with the next version of the program. Rather than gambling away the planning time, try some exploratory tests – whatever comes to mind.” More recently, Exploratory Testing has come into its own as a testing paradigm, as summarized on Brian Marick’s superb Testing Craft web site: http://www.testingcraft.com/exploratory.html. In this excerpt, he lists the various elements associated with Exploratory Testing: • An interweaving of test design and test execution. This is in contrast to a process in which the tests are all designed first, then run later. • A notion that the tester is learning about the product by testing it. • An emphasis on creativity and spontaneity. • (Sometimes) An expectation that exploratory testing will change the allocation of effort. Testing thus precedes some parts of test planning. James Bach has published a formal procedure for Exploratory Testing, available on the Testing Craft site mentioned above. One way he characterizes the Exploratory paradigm is that “the outcome of this test influences the design of the next test.” James also offers a seminar in Exploratory Testing, described on his website: http://www.satisfice.com/seminars.htm. So how does the notion of an ad hoc test relate to Exploratory Testing? We assert that ad hoc testing is a special case of Exploratory Testing. In the course of doing Exploratory Testing, many of the test cases will be ad hoc (one-off tests), but some cases will not be. One way to distinguish between the two is to look at the notes associated with a given exploratory test. In general, exploratory tests have little or no formal documentation, but result in more informal notes. If the notes are detailed enough that the test could be rerun by reading them, then that is less likely to be an ad hoc test. Conversely, if there are no notes for a given exploratory test, or if the notes are directed more at guiding the testing effort than at reproducing the test, then this is almost surely an ad hoc test. On the whole, software managers tend to view ad hoc testing skeptically. The notion of highly paid testers consuming precious time running one-off tests, and then discarding them, is apt to make managers squirm. However, skilled exploratory testers have little trouble justifying their time spent on ad hoc cases. The quantity and severity of defects unearthed using this approach can be astounding. In this paper, we’ll attempt to provide a useful context for this testing paradigm. Ad hoc tests versus Regression tests One aim of this paper is to propose a more precise definition for an ad hoc test. A standard definition for ad hoc is “for the particular end or case at hand without consideration of wider application” (Miriam-Webster, http://www.m-w.com/). Based on this, we will describe an ad hoc test as an exploratory case that you expect to run only once, unless it uncovers a defect. We're aware that some people object to this definition of ad hoc testing, so if you’re in that camp, please feel free to substitute the term “one-off” for all instances of “ad hoc” in this paper. We prefer the term ad hoc, primarily for aesthetic reasons, but will use the two terms interchangeably in this paper. We'll try to put ad hoc testing into perspective here, by comparing it with other forms of testing. In particular, each method has strengths and weaknesses in the critical dimensions of defect finding power and confidence building. A primary goal of ad hoc testing is to uncover new defects in the product. In the hands of a skilled tester, it can be highly effective at discovering such problems. As a confidence builder, ad hoc testing is relatively weak, compared with formal regression testing, which can be a powerful confidence builder, especially if the breadth and depth of the coverage is demonstrably high. The term regression testing has come to mean many different things, depending on the context in which it is used. For example, Cem Kaner has described these basic types of regression testing: • Bug regression try to prove that a newly fixed bug was not fixed. • Old bugs regression Try to prove that a recent code (or data) change broke old fixes. • Side effect regression Try to prove that a recent code change broke some other part of the program. In the descriptions above, notice the emphasis on proving there are defects in the code, thus putting a premium on the defect finding power of regression testing. In contrast, consider this definition from the whatis online technical encyclopedia (http://www.whatis.com/): “Regression testing is the process of testing changes to computer programs to make sure that the older programming still works with the new changes. Regression testing is a normal part of the program development process and, in larger companies, is done by code testing specialists. Test department coders develop code test scenarios and exercises that will test new units of code after they have been written. These test cases form what becomes the test bucket. Before a new version of a software product is released, the old test cases are run against the new version to make sure that all the old capabilities still work. The reason they might not work is because changing or adding new code to a program can easily introduce errors into code that is not intended to be changed.” In the definition above from whatis, notice the emphasis is placed on ensuring that the program is working, thus putting a premium on the confidence building aspect of regression testing. For the purpose of this paper, we’re going to use the definition from whatis, not because we think it’s a better approach than putting the emphasis on defect finding, but because we think this is closer to how the term regression testing is more commonly used today. We’re also going to expand on this definition: implicit in the concept of creating a regression suite is the notion that the tests will be rerun periodically. Otherwise, there is little value in spending time creating the suite in the first place. Furthermore, many companies opt to automate their regression suite, specifically so that the tests can be rerun faster, less expensively, and more often. In this respect, regression testing is the opposite of ad hoc testing: an implicit yet important attribute of a regression test is that you expect to rerun it, in contrast to an ad hoc test, which you expect to run only once. Although they are at the opposite ends of this spectrum, regression tests and ad hoc tests complement each other in remarkable ways. Ironically, when you uncover a defect with either form of testing, you'll be drawn toward the opposite end of the spectrum. When you find a defect with a regression test, you'll often need to use ad hoc methods to analyze and isolate the defect, and to narrow down the steps to reproduce it. You may also want to explore “around” the defect to determine if there are related problems. On the other hand, when you find a defect with an ad hoc test, you'll probably document it in your defect tracking system, thereby turning it into a regression test, for verification after the defect is fixed. Let's consider some other specific types of testing along this spectrum. For the sake of visualization, we’ll place fully repeatable regression tests on the left end of the spectrum, and ad hoc tests on the right end. For any given method used by a tester, its location along this spectrum will vary, depending on the methods being used by the tester. For example, performance testing and release testing have a dominant regression testing component, so we would place them along the left side of the spectrum. Exploratory testing and User Scenario testing have a dominant ad hoc component, so we would place them somewhere on the right side of the spectrum. We’re convinced that most software testing projects will benefit from an intelligent combination of the various forms of testing. Rare is the project that can be tested optimally by regression testing or by ad hoc testing alone, but by finding the ideal proportions for each project, you'll have a better balance between defect finding and confidence building. The Metaphorical Bridge The spectrum described above can also be pictured as a bridge between formal and less formal testing paradigms. This metaphor has the advantage of suggesting motion across the bridge in both directions. For example, you can bring regression tests across the bridge into the ad hoc domain for more extensive testing. Conversely, you can bring ad hoc tests that you choose to rerun across the bridge in the other direction, for integration with the regression suite. We’ll expand this bridge metaphor more fully in the section on Improvisational Testing, below. A Bridge Between Paradigms